Factors Associated With Nonadherence to Lung Cancer Screening Across Multiple Screening Time Points

This cohort study investigates factors associated with patient nonadherence to lung cancer screening recommendations across multiple time points.


Supplemental Methods
Identification of screen-eligible patients: The primary care providers or pulmonologists determined if a patient was eligible to participate in lung cancer screening (LCS), then referred them to the LCS program. Health professionals in the Screening Program provided shared decision making (including smoking cessation counseling) if this had not been documented in the electronic medical record.
Intervention for adherence: After each screen, the patients would receive reminders for the follow-up examination from two sources, 1) their primary care providers or pulmonologists and 2) staff of our LCS program. However, patients who decided not to come back for screening were not contacted to record the reasons for non-adherence.
Documentation of time data: The completion date of each LDCT was retrieved from the Integrated Diagnostic Lung database. If a follow-up screen was completed, then the difference between the two LDCT dates was used to determine adherence; if a follow-up screen was not completed, then we would wait 15 (Lung-RADS 1 and 2), 9 (Lung-RADS 3), 5 (Lung-RADS 4A), and 3 (Lung-RADS 4B/X) months before categorizing the patient as non-adherent.
Determination of screening time points: The three screening time points were the actual dates of the patient's first three low-dose CT screening examinations. T0: baseline/first screen; T1: second screen; T2: third screen.

Supplemental Results
Rate of non-adherence among patients with lung cancer: Among patients diagnosed with lung cancer, 11% (9/81) had not adhered to screening recommendation.

Methods
We trained and evaluated five machine learning models to predict patient non-adherence to baseline Lung-RADS recommendations (see eFigure 2 in the Supplement). The inputs into the models were significant (i.e., z test, two-sided p-value<0.05) baseline predictors from Experiment 1. This experiment aimed to validate whether these predictors could correctly classify patients who were non-adherent to baseline Lung-RADS recommendations. Logistic regression, random forest, support vector machine, naïve Bayes, and XGBoost were trained using data without missing values in all input variables. Naïve Bayes was also trained using data with missing values left as-is. We used nested 10-fold cross-validation, repeated five times to select the best models based on the primary evaluation metric, recall/sensitivity, in the validation sets, which reflected the model's ability to identify truly non-adherent patients correctly. The best-performing model was then retrained and tested on the hold-out test cases (n=278 from data with no missing values in significant predictors from Experiment 1). Secondary evaluation metrics included precision/positive predictive value (PPV), accuracy, and area under the receiver operating characteristic curve (ROC-AUC).

Results
Among 2496 eligible patients, 278 with no missing values in significant predictors from Experiment 1 were used as the hold-out test set. Of the remaining 2218 patients, 300 patients had missing values in some significant predictors from Experiment 1, leaving 1918 patients with no missing values in the significant predictors from Experiment 1. 2218 (with missing values) and 1918 (without missing values) patients were used for cross-validation (see eFigure 2 in the Supplement). The inputs into the machine learning models were the six significant baseline predictors from Experiment 1. Model performance on the validation sets of the five machine learning models is shown in eTable 3. Most models achieved greater than 90% recall/sensitivity and similar performance in other evaluation metrics. The final retrained logistic regression model achieved recall/sensitivity: 0.939, precision/PPV: 0.712, accuracy: 0.716, and ROC-AUC: 0.667 on the hold-out test cases.

Discussion
We show that machine learning models trained on significant predictors identified in Experiment 1 can capture most nonadherent patients (i.e., high recall/sensitivity), only missing 6% of non-adherent patients. Given that some predictors may not be routinely collected in medical records (e.g., income), the model that handles missing values (i.e., naïve Bayes) is useful for making classification when values of certain variables are missing. We should note that the analysis was influenced by the screening population that is seen at our institution; other institutions may identify specific predictors that affect adherence in their lung cancer screening population.

Limitation
Despite a high recall/sensitivity, the accuracy of the prediction models was around 70%, resulting in some patients who are likely to be adherent in practice being misidentified as having a high risk of non-adherence. In a targeted approach to adherence interventions, these individuals may receive unnecessary outreach; in this scenario, the negative impact is minimal to the patient but may divert critical resources away from other essential services.